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ABSTRACT 

An evaluation design created to provide information 
for a multiple- pro ject program of educational experimentation is 
presented. Project SEED (State Experimentation in Educational 
Development) became an official education effort in North Carolina on 
July ^, 1971, Model for the program was that of the State's ESEA 
Title III program with certain exceptions. The 19 projects selected 
were funded to conduct innovative programs in 13 different priority 
areas. They were designed to be carried out in elementary, junior and 
senior high schools^ an entire LEA and in two regional centers which 
provided services for 10 or more LEAs • A review of the SEED program 
revealed a need for two separate evaluations and for information to 
identify possible weaknesses. It is concluded that the evaluation 
design implemented clearly demonstrates the feasibility of utilizing 
a model of formative and s.ummative evaluation in multiple-pro ject 
programs • (CK) 
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Introduction 

When an agency or group is given the task of evaluating a multiple Dro.iect 
program, it is faced with unique and often complex strategies for the evalua- 
tion design, data collection and synthesis of results. The process becomes 
more complicated when the projects are dissimilar and the collection of across 
project data is unsuited to the needs of the audiences served. Add to this 
a desire for both formative and summative information collected primarily by 
local project personnel with negligible evaluation experience and we come to 
ths situation which faced the Division of Research in the North Carolina 
Department of Public Instruction in October, 1971. 

It is the purpose of this paper to present an evaluation design which was 

created to provide information for a multiple project program of educational 

experimentation a design which encompassed: (1) the evaluation needs of 

the State Agency and 19 uniquely individual projects; (2) a model for the 

collection of both formative and summative evaluation information; and (3) the 

involvement of local project personnel in the evaluation process. In addition, 

the paper contains information concerning the implementation and implica- 
tions of the design. 

THE PROGRAM 

Project SEED (State Experimentation in Educational Development) became 
an official education effort in North Carolina on July 1, 1971. Monies were 
appropriated by the State exclusively to promote experimentation, innovation 
and education growth and development in the LEAs. Six hundred thousand dollars 
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was appropriated for Implementation of experimental projects during the 1971-73 

biennium. Although this amount was less than requested, it did allow for initial 

program development in 19 of the State's 152 administrative units. 

The model adopted for the implementation of the new program was basically 

that of the State's ESEA Title III program with certain exceptions due to the 

low level of funding and "the commitment of the State or provide the widest 

distribution of experimental experiences to the 152 LEAs. As a result, priority 

was given to those plans which: 

. would be carried out in a school which did not have on-going 
Middle School Occupational Programs, ESEA Title III orojects 
or State approved kindergartens; 

r focused upon one of the eleven State priority areas for 
experimentation utilized in the select^ion of ESEA Title III 
projects. (Consideration was to be given, however, to any 
other area which would set a new direction for North Carolina 
education.) The areas were: 

Community Schools Performance Accountability 

Differentiated Staffing Reading 
Drug Use/Abuse Education School -Community Relations 

Economic Education Value Development 

Management and Leadership Year-Round School 

Development 

. initiated new and innovative programs, y^ather than perpetuated 
old ESEA Title III, CSIP or other developmental programs; and 

. allocated the bulk of the money for the program, and not for 
its administration. 

The nineteen projects selected varied greatly in the area chosen for 

experimentation, the strategies employed to experiment in any of the given areas, 

and in the level chosen for both implementation and management of the project. 

The projects were funded to conduct innovative programs in 13 different priority 

areas. In all cases where the priority area, was the same, the strategies and/or 

level chosen for implementation differed. They were designed to be carried 

cut in elementary, junior and senior high schools, an entire LEA and in two 

regional centers which provided services for 10 or more LEAs. Persons with major 
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responsibilities for the implementation of the projects included state agency 
personnel in the regional centers, local assistant superintendents, suoervi- 
sors, principals and teachers. Each project was, therefore, unique in both 
approach and method of imp,lementation. Table I presents a summary of the 
projects selected. 

The first major activity for local project personnel was the exoansion 
of the rnini-proposal into a more uniform and comprehensive proposal, or project 
plan. They, with the assistance of the State SEED staff and selected State 
education agency consultants with expertise in the chosen area of experimental 
tion, developed an expanded proposal which included (1) a statistical section: 

(2) a description of local needs and the problem area selected for the project; 

(3) the general and specific objectives of the project with accompanying 
strategies, and (4) the proposed budget. 

This method of writing project proposals had significance for the evalua- 
tion design selected because: (1) all proposals were written in the same 
format allowing for synthesis of across project information; (2) it provided 
the project personnel, who had varied levels of experience in projects of 
this kind* with a workable plan for the implementation and thus the evaluation 
of their project; and (3) it gave the State SEED staff, evaluation consultants, 
and local project personnel a similar frafnework on which to base their partic- 
ular activities. 

The projects became operational in January, 1972, after each of the 
proposals had been approved. 
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THE EVALUATION 

The SEED staff was interested in information regarding the effectiveness 
of this new program. To secure this information, the State Agency's Division 
of Deve]opmenr> or which the SEED program is a component, contracted to an 
independent division of the State Agency, the Division of Research, for an 
evaluation. Five percent of the program budget, or $30,000, was orovided to 
f-inance the two year evaluation- An evaluation consultant was hired in 
October, 1971, to: (1) develop the evaluation plan; (2) provide evaluation 
assistance to ind'n/TdUal projects; and (3J provide evaluation information to 
appropriate decuvon makers. 

OBJECTIVES OF THE DESIGN 

A review of. the SEED program and its information needs revealed the 

following evaluation requirements: 

. two separate evaluations were needed: one for the State 
agency of the entire SEED program and individual evaluations 
for each of the nineteen local projects 

. information was needed by both groups not only at the end 
of the progrdm but during the implementation of the program 
to identify possible weaknesses which could be modified to 
enhance program success. 

An evaluation design was thereby needed which would provide two types of 
evaluation, evaluation of progress and outcomes, for two audiences, the State 
agency and the individual LEAs. These considerations led to the selection of 
a model utilizing the concepts of formative and summative evaluation. 

Scriven^ in his now almost classic article, describes formative evalua- 
tion as a process of discovering deficiencies and successes in the intermediate 
version of a new curriculum. Summative evaluation, as he defines it, is 

^Michael S< Scriven, "The Methodology of Evaluation" in Perspectives of 
Curriculum Evaluation (AERA Monograph Series on Curriculum Evaluation, 
No, 1). Chicago: Rand McNally and Company, 1967. 

ERIC 
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concerned with evaluating the effects of a whole teacher-curriculum oackage. 

Transferring these concepts from curriculum to project evaluation led to the 

development of the following definitions. 

Formative evaluation is the process of proi^iding information at 
intervals in the project progress so that additions ^ delations ^ 
or modifications oan be made to maximize project success. 

Siamative evaluation refers to a process of providing information 
concerning project outcomes so that decisions can he made 
regarding the continuation^ rejection or modification of a project. 

The formative/summati ve evaluation concept, however appealing, was not 
easy to implement. The concept was relatively new and a review of the litera- 
ture revealed theoretical positions rather than implementation techniques. 
Therefore, not only the design but techniques for de^kign implementation had to 
be developed. Secondly, few, if any, of the persons associated with the SEED 
program were aware of the concept of formative and summative evaluation and even 
fewer were fully cognizant of its implications for project operation* It was 
necessary to sell the concept in order to establish an awareness of and commit- 
ment to the design. Finally, the placement of persons with the eV(aluation 
expertise needed to carry out a formati ve/summati ve evaluation in each of the 
project sites was not possible due to the limited funds available. The level 
of evaluation expertise in the Division of Research was sufficient to develop 
strategies for the implementation of a formati ve/summati ve evaluation model. 
There remained the problems of awareness of, commitment to, and implementation 
of the concept. The only practical and desirable solution was the direct 
involvement of local project personnel in the evaluation process. 

Effective local involvement in the evaluation of multiple project programs 
depends on the ability of project personnel to carry out the required activities. 
Since there was great divergence in the evaluation awareness and competence of 
local project SEED personnel, an additional component of evaluation training 
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was added to the evaluation design. Inclusion of this training would provide 
personnel with the skills necessary to conduct the major oortion of a formative/ 
suinmaiive evaluation of their project and would, hopefully, demonstrate the 
significance of evaluation utilization in their educational decision making. 

In summary, the evaluation plan for Project SEED was designed to meet the 
following objectives: 

Ural 

K To provide information to project directors on a regularly 
scheduled basis concerning progress of their individual SEED 
projects. (Formative) 

2. To provide information to the local education policy maKers 
upon which to base decisions regarding continuation, rejection, 
or modification of their individual SEED projects. (Sumrnative) 

State 

1. To provide information after one year's operation to the bDPi 
concerning the progress of the total SEED project. (Fortnati vej 

2. To provide information to the SDPI upon which to base decisions 
regarding continuation, rejection, or modification of the 
total SEED program, (Sumrnative) 

Training 

1. To provide local project personnel with expertise to carry out 
most of the major evaluation activities required by their SEED 
project. 

The following sections outline in further detail the plan as it was designed 
and implemented. 



THE DESIGN 

The Local Project Evaluation Design 

Formative ■ Two major objectives were conceived as being important in the 
development of a formative evaluation design for the individual projects: 
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(1) To detect weaknesses in the project p^an prior to implementation 
so that adaed attention could be devoted to those areas, 

(2) To obtain information during the project's operation concerning 
the success of project implementation so that necessary modifica- 
tion could he made. 

To detect weaknesses prior to project impleji^entation required a thorough 
review of the project as it had been planned. Project personnel, with assis- 
tance of a State Agency evaluation consultant completed a form designated the 
"Formative Evaluation Plan'' which contained a listing of the specific objec- 
tives of the project, their accompanying strategies anH tasks deemed critical 
to the accomplishment of the strategies. In columns beside each of the pre- 
ceding were listed the persons responsible for the activity, the duration of 
the activity, and an identification of those activities to be evaluated. The 
process of completing this form gave the project director and consultant an 
opportunity to review in detail with one or more members of the project staff 
the planned implementation of the project. Problem areas which were discovered 
during thij review were either corrected or noted for additional attention 
during implementation. The completed form provided the project personnel with 
a concise statement of the entire project from beginning to end which could be 
used for project implementation, management, and evaluation. 

The following factors were identified as important in obtaining informa- 
tion relative to the success of project implementation: 

1. Is the project operating on schedule? 

2. Are the available resources (personnel, facilities, finances, 
etc.) adequate for project operation? 

3. What activities are creating problems in the operation of the 
project? 

4. What activities can be easily assimilated into similar and/or 
regular school programs? 

5. Is the project progressing toward the achievement of its objectives? 

What activities should be added, deleted, or modified to increase 
the effectiveness of the project? 

ERIC 
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Several alternatives were considered to provide this information. In 
the final analysis, the decision was made to select i technique which would 
provide the State and the local project personnel with information concerning 
each of the project activities so that needed modifications could be made 
during implementation and so that activities which had been successful could 
easily be identified at the conclusion of the project. Since the strategies 
listed under each objective were, in reality, the major activities which would 
be undertaken, the decision was made tc evaluate each of the strategies in 
relation to the objectives for which they were written. A method was devised 
in which the project personnel, all of whom had major responsibilities other 
than SEED, could easily obtain this information. 

A strategy evaluation checklist was designed to provide the information 
asked in the questions above. As each strategy was completed, it was analyzed 
using this checklist by local project personnel. Necessary changes which 
would enhance the effect of the project were then made on the basis of this 
review* 

Summati ve - Rather than assess the accomplishment of global objectives 

m 

Or measure several isolated variables, it was determined that the summati ve 
evaluation for each of the projects consist of an analysis of the accomplishment 
of each of the project's specific objectives. This approach would allow the 
LEA to analyze the success of each of the project components, thereby providing 
specific information upon which to base decisions regarding adoption, rejection, 
or modification of part or all of the project plan. Furthermore, the results 
would provide information to the State Agency concerning the project's success 
and promising activities which could be carried out in other LEAs. 

As with the formative evaluation, the project personnel, with the assistance 
of an evaluation consultant, completed an outline called the "Summati ve Fvaluation 
Plan/' This plan included a listing of the specific objective, information 
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concerning the sample from which the data would be collected, the instrument 
to be used in data collection, the type of analysis to be used on tne nata 
and the date by which the analysis was to be completed. By comoletinq the 
plan, it was anticipated that project personnel would become more tdmiliar 
with the techniques of project evaluation. The plan, itself, oroviaed the 
project staff with a detailed outline of the activities to be accomnlisnen 
to evaluate each of their objectives. 

The development of the formative and surmnative evaluation oians trovided 
project personnel with a "tailor-made" guide for both the operation and evdlua- 
tion of their project. The decisions determining the data to be cciiected 
were made by local personnel, resulting in evaluation information that was 
directly related to local needs. This relationship was built into the design 
in the requirement tbst each evaluation activity be directly associated with 
a project goal or activity. The plan for the collection of formative evalua- 
tion results provided local personnel with a relatively quick and easy method 
of analyzing each of the projects' activities as they occurred. More impor- 
tantly, the plan permitted change in the project proposal based on information 
concerning the degree to which activities succeeded in contributing to the 
achievement of project objectives. As a result, the total operation of the 
projects became more fluid and less regimented as activities which clearly 
did not contribute to project success could be discarded and activities which 
were successful would be amplified. 

The State Level Evaluation Plan 

No formal formative and summative plans were adopted at the State level 
as had been done in the individual projects since the State plan was not written 
as a proposal with accompanying objectives and strategies. However, both types 
of evaluation were employed in the design for the total program. 



Formati ve 

ormative evaluation activities centered on regular and irregular forms 
f >n\j\}t from various sources. These inputs, which included regular meetings 
between the SEED staff and evaluation consultant, provided information upon 
which the SEED program staff could base decisions regarding the policies 
which guided the implementation of the program. . The interim evaluation report, 
prepared after the first year of program operation, presented a summary of 
the formative evaluation results. The report contained an overview of the 
progress of each of the 19 individual projects, a summary of the State SEED 
activities to date, and recommendations for the second year's operation, 
Information reported was compiled from evaluation reports submitted to the 
evaluation consultant by the local project personnel, available records on 
file of SEED activities, interviews with the SEED staff and information 
gleaned from other available resources. 

Summati ve 

The purpose of a summative evaluation of the total program is to provide 
information for the State level decision makers concerning the success of the 
implementation of the program. In addition, those groups which are concerned 
with developmental activities, which in this .case includes both the LEAs and 
the State Education Agency, require documentation for continuation of a program 
and for potential adopters. To accomplish this, the summative evaluation design 
for the total SEED program contains the following components: 

. a review of the SEED program as it was designed and implemented 

. an evaluation of each of the 19 local projects 

. synthesis of relevant across-project data 

c results of a survey designed to determine the impact of SEED 
in the local unit and the success of its implementation 

. documentation for decision makers at various levels to 
utilize in planning for continuation or adoption of the 
^ Successful components of the program. 

ERIC 
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Th1s information will be presented in three documents: a State level evalua- 
tion report, evaluation reports for each of the 19 projects and an "Adopter's 
Fi'Ie" for each of the projects. These reports will provide information upon 
which to base both State and local decisions related to the program and assist 
in the development of an awareness of SEED and the individual projects^ to 
potential utilizers of the findings across the State. 
TRAINING 

As mentioned in a previous section, evaluation training for local project 
personnel was a vital part of the evaluation design. The four training sessions, 
conducted during the two year period of SEED operation, were designed to focus 
on the needs of local project personnel as determined by the types of evalua- 
tion activities outlined in the collective summative evaluation plans of all 
of the projects. A review of the summative plans, along with consideration 
of the training needs required to implement the total evaluation plan, led to 
the selection of the following topics for each of the four sessions: 

I. Formative and Summative Evaluation 

Development of Local Formative and Summative Evaluation Plans 

II. The Importance of Evaluation at the Local Level 
Selecting a Random Sample 
Organization of Record Keeping Activities 
Questionnaire Design 

III. Testing 

Evaluation Design 

IV. Data Presentation 

Writing of Local Evaluation Reports 

Papers relating to these topics were contained with additional evaluation 

information in a Handbook of Evaluation Techniques^ which was given to each of 

the participants . 

2 

^Tanya M. Kniefel, Handbook of Evaluation Techniques , Raleigh, North Carolina: 
Division of Research, North Carolina State Department of Public Instruction, 
April, 1972. 




-13- 



Since all project directors and most of the other personnel working in 
the project had other major responsibilities and obligations, efficiency in 
the time spent for training was another important consideration. For those 
sessions (the first and last) that dealt with the development of a product 
(the evaluation plans and evaluation report) individual assistance was pro- 
vid6d to each project by a Division of Research evaluation consultant in 
Raleigh in order to minimize the time required to develop the product at the 
local level. The remaining sessions were held in a/eas more centrally 
located to the projects in the western and eastern parts of the State in 
order to minimize travel time and maximize attendance, length of time avail- 
able for training, and opportunity for open discussion. 

Although evaluation training was specifically designed to provide local 
project personnel with the skills necessary for successful involvement in 
the evaluation of their projects, the sessions provided an opportunity for 
additional desirable activities. Project personnel were able to get together 
with others in the program and exchange successes and problems. Quite often 
people found that their problems were not unique and that they could work 
together toward solutions. . The sessions also provided an important opportu- 
nity for the establishment of rapport between the local project personnel 
and the Project SEED evaluator. This rapport was critical to the achievement 
of the cooperation needed between the State and local education agencies to 
adequately conduct the eval uations of the local projects. Finally, the 
sessions provided a forum for the presentation and discussion of new ideas and 
practices relating to both the projects and their evaluation. 



^ CONCLUSIONS AND IMPLICATIONS 

Several writers such as Stufflebeam et al have proposed specific criteria 
to which evaluations must conform to be considered "accentable. " Basically, 
these criteria seem to indicate that the following two questions be satis- 
factorily' answered in order to adequately justify utilization of the desiqn. 

. To what extent does the design provide valid, relevant infor- 
mation to the decision maker? 

. To what extent is the implementation of the design feasible 
in terms of cost, materials and equipment required, oersonnel , 
etc? 

It is the purpose of the remainder of this paper to relate the desiqn 
presented to the criteria represented in these questions. The reader may then 
judge for himself if the answers provided are satisfactory. 

Every attempt was made to gather valid and reliable data for each of the 
evaluations. The selection of data to be collected for the individual projects 
was made collectively by local project personnel to ensure the collection of 
data related to local needs, subject area specialists to ensure the selection 
of current, appropriate assessment techniques, and an evaluation consultant 
to ensure the appropriateness of the data collection techniques. Individual 
assistance provided by a trained evaluator andevaluation training sessions 
were the vehicles utilized to induce validity and objectivity into the imple- 
mentation of the evaluation activities. 

Fundirig restrictions prohibited the utilization of strategies designed 
to control the effect of possible biases and loss of objectivity introduced 
when persons responsible for evaluation were also responsible for project 
implementation and success. An audit of the evaluation by an independent 
third party, impossible due to limited funding, was greatly desired and sorely 
missed. Additional attention was, therefore, given to each evaluation activity 
in order to limit the effect of these biases. 

-^Daniel L. Stufflebeam, et^ al . Educational Evaluation and Decision Making , 
Itasca, 111,, F. E. PeacocIT Publ ishers , Inc., 1971. 
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A final area of concern in the design was the dependence of the formative 
and summative evaluation on the quality of the written proposal. Efforts to 
control this possible limitation were made by the inclusion of proposal writing 
as a major program activity. Input in the development of the proposal from 
local project personnel, SEED staff consultants with expertise in the develop- 
ment of innovative projects, and subject area specialists was designed to 
increase validity of both the objectives of the project and the activities 
chosen to achieve the objectives. 

Relevance of the evaluation to the client served was a key consideration 
in the development of the design and the procedures for reporting the results. 
Continual communication, both informally and as activities built into the design, 
permitted the collection of information directly related to both State and 
local needs. This component appeared to be crucial to the establishment of 
commitment to the design and cooperation in its implementation. A concerted 
effort was also made to provide reporting procedures which could be utilized 
by a wide variety of audiences. Assistance provided by State SEED and local 
project personnel in both the planning and development of the final reports 
is designed to ensure the production of documents suited to the various levels 
of decision making in the State, 

The feasibility of implementing the design was certainly demonstrated by 
the fact that all of the planned activities were acconplished with minimal staff 
(one evaluation consultant with the assistance of local project personnel and 
four State SEED consultants) and limited funds ($30,000 for two years). Several 
factors seemed to have contributed significantly to the implementation of a 
design conducted under such monetary and personnel restrictions. First, every- 
one associated with the evaluation worked very hard, long hours beyond their 
regularly scheduled responsibilities. 
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A second factor may have been related to the low level of funding itself. 
Most of the projects were necessarily, becau5;e of budget, conducted in a single 
school setting. As a result, the samples from which the data were collected 
were not excessively large (the largest was N=700) reducing the time required 
for data collection and analysis. Finally, analysis of results was greatly 
facilitated by the availability of the services of statistical analysts and 
a computer in the State Agency. 

Local involvement in the evaluation process was such a key comnonent to 
the implementation and success of the design that special note should be taken 
of its contribution. As noted in the earlier description of the SEED nrogram, 
there were few if any similarities between the persons responsible for the 
local SEED projects. In many cases, they had never actively participated in 
such a project. It was, therefore, no surprise that their first reaction to 
formative and summative evaluation and its implications for them as they con- 
ducted a project was one of hesitation and bewilderment. For many reasons, 
perhaps the excitement of Implementing an Innovative project, the 
independence In project operation permitted as a result of formative evalua- 
tion, or the assistance provided in the evaluation activities, local oroject 
personnel cooperated to the fullest in every activity introduced to them. 
Much to the surprise of those responsible for the evaluation, they became the 
design's staunchest supporters extolling its virtues at every possible opportu- 
nity. Surely, this attitude and the diligent manner in which they conducted 
evaluation activities contributed greatly to the success of the overall design. 

In summary, those of us who have been associated with the development and 
implementation of this evaluation design feel that it has definite imnllcations 
for future evaluation efforts. The significance of the successful imolementatlon 
of the design lies in the fact that it clearly demonstrates the feasibility 

EKLC 
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of utilizing a model of formative and summative evaluation in multiple- 
project programs- It further supports the premise that, with training, local 
project personnel can and should be tivolved in the evaluation process. Finally, 
even though it is still in an experimental form, the design has provided us 
with a beginning, a place to start, as we attempt to enrornn^^vtg relevant^ 
meaningful evaluation results into the educational policy decision makinq 
process in North Carolina. 
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